Multiple Classifier Combination through Ensembles and Data Generation

An ensemble of classifiers consists of a set of individually trained classifiers whose predictions are combined when classifying new instances. The resulting ensemble is generally more accurate than the individual classifiers it consists of. In particular, one of the most popular ensemble methods, the Boosting approach, improves the predictive performance of weak classifiers, which can achieve only an error rate of at least slightly better than random guessing. This improvement is achieved by focusing on examples that are hard to be correctly classified, i.e. the so-called hard examples. The Boosting approach has been applied to many real world domains, including medicine, bioinformatics, fraud detection and text classification, amongst others, with great success. However, in many domains, Boosting algorithms frequently suffer from overemphasizing these hard examples. That is, the Boosting algorithm puts too much effort on the hard examples, thus leading to poor training and test set accuracies. Moreover, the knowledge acquired from such hard examples may be insufficient to improve the overall accuracy of the ensemble. Also, when applied to two-class domains with imbalanced class frequencies, where the number of examples of one (majority) class is much higher than the other (minority), a traditional Boosting algorithm tends to produce a high predictive accuracy over the majority class, but poor accuracy over the minority class. This is due to a number of factors. Firstly, the learning algorithms tend to ignore small classes while concentrating on classifying the large ones accurately (i.e. the so-called learning bias). Secondly, the weight updating mechanism of the Boosting algorithms may deteriorate the learning bias. Thirdly, examples from the minority class may provide insufficient knowledge for learning. In addition, the few examples from the minority class are much easier to be over-emphasized by the Boosting algorithms. This thesis introduces new techniques to address the above-mentioned two problems. Two approaches, namely the DataBoost and DataBoost-IM algorithms, are provided to extend Boosting algorithms’ predictive performance. The DataBoost algorithm is designed to assist Boosting algorithms to avoid overemphasizing hard examples. In the DataBoost algorithm, new synthetic data with bias information towards hard examples are added to the original training set when training the component classifiers. The DataBoost approach was evaluated against ten data sets, using both decision trees and neural networks as base classifiers. The experiments show promising results, in terms of overall accuracy when compared to a standard benchmarking Boosting algorithm. The DataBoost-IM algorithm is developed to learn from two-class imbalanced data sets. In the DataBoost-IM approach, the class frequencies and the total weights against different classes within the ensemble’s training set are rebalanced by adding new synthetic data. The DataBoost-IM method was evaluated, in terms of the F-measures, G-mean and overall accuracy, against seventeen highly and moderately imbalanced data sets using decision trees as base classifiers. The experimental results show that the DataBoost-IM method compares well in comparison with a base classifier, a standard benchmarking Boosting algorithm and three advanced Boosting-based algorithms for imbalanced data sets. Results indicate that the DataBoost-IM approach does not sacrifice one class in favor of the other, but produces high predictions against both minority and majority classes.